Skip to content

test: 新增 Session/Memory/Summary 回放一致性测试框架#115

Open
xyxhhhhh wants to merge 1 commit into
trpc-group:mainfrom
xyxhhhhh:issue-89-session-memory-replay-consistency
Open

test: 新增 Session/Memory/Summary 回放一致性测试框架#115
xyxhhhhh wants to merge 1 commit into
trpc-group:mainfrom
xyxhhhhh:issue-89-session-memory-replay-consistency

Conversation

@xyxhhhhh

@xyxhhhhh xyxhhhhh commented Jul 3, 2026

Copy link
Copy Markdown

概述

Closes #89

为 Session / Memory / Summary 引入回放一致性测试框架,用于验证会话历史、内存记录和摘要内容在不同存储后端、不同 replay 场景下是否保持稳定、可解释、可审计。

完成内容

  • 新增 Session / Memory / Summary replay consistency 测试框架
  • 覆盖 in_memorysqlite 两类后端
  • 新增 18 条 JSONL replay case,覆盖正常、边界和异常场景
  • 支持 lightweight replay mode,避免依赖外部 LLM 服务
  • 生成结构化 diff report,便于审查和回归定位
  • 新增中文设计文档,说明测试目标、覆盖范围、报告格式和已知边界

覆盖场景

当前样例覆盖:

  • 单轮会话回放
  • 多轮会话回放
  • Memory 写入和读取一致性
  • Summary 生成后的状态回放
  • Session 与 Memory 组合场景
  • Summary 与历史消息组合场景
  • 空消息、重复消息、长文本等边界输入
  • 不同后端之间的 replay 对比
  • 异常或不可恢复输入的结构化记录

主要文件

  • tests/sessions/test_replay_consistency.py:replay consistency 测试主逻辑,包含后端参数化、diff 分类、断言、性能和误报率检查
  • tests/sessions/replay_cases/session_memory_summary_replay_cases.jsonl:18 条公开 replay 样例,可继续追加扩展
  • session_memory_summary_diff_report.json:当前样例的结构化回放报告,记录 coverage、case 结果、diff 详情和错误统计
  • docs/mkdocs/zh/session_replay_consistency.md:中文设计说明,解释 replay 目标、报告结构、扩展方式和限制

验收映射

  • 可重复 replay:通过固定 JSONL case 和 deterministic lightweight mode 实现
  • 覆盖 Session / Memory / Summary:测试用例覆盖三类状态及组合场景
  • 支持多后端:当前覆盖 in_memorysqlite
  • 输出结构化报告:生成 session_memory_summary_diff_report.json
  • 可定位差异:报告中包含 case id、backend、diff 类型和具体结果
  • 可持续扩展:新增 case 只需追加 JSONL 样例
  • 避免外部服务依赖:lightweight mode 不依赖真实 LLM 调用
  • 文档说明:新增中文设计文档解释机制和边界

报告结果摘要

当前 session_memory_summary_diff_report.json 显示:

  • case_count: 18
  • backend_error_count: 0
  • normal_diff_count: 0
  • normal_false_positive_rate: 0.0
  • mode.lightweight: true
  • attempted backends: in_memory, sqlite
  • snapshot backends: in_memory, sqlite

本地验证

python -m pytest tests/sessions/test_replay_consistency.py -q
git diff --check
python -m json.tool session_memory_summary_diff_report.json

本地结果:

  • tests/sessions/test_replay_consistency.py:64 passed
  • git diff --check:通过
  • session_memory_summary_diff_report.json:JSON 格式合法

设计边界

本次实现是提供一致性测试和审计能力,不改变默认运行时行为。对于依赖模型生成内容的场景,使用 lightweight replay mode 固定可比较输出,重点验证Session / Memory / Summary 状态流转、存储读写和报告生成是否稳定。

@xyxhhhhh xyxhhhhh force-pushed the issue-89-session-memory-replay-consistency branch 2 times, most recently from f7c2d6c to 0e8ef83 Compare July 3, 2026 12:20
@xyxhhhhh xyxhhhhh force-pushed the issue-89-session-memory-replay-consistency branch from 0e8ef83 to bc6e1d0 Compare July 3, 2026 12:32
@codecov

codecov Bot commented Jul 3, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
⚠️ Please upload report for BASE (main@73655ab). Learn more about missing BASE report.

Additional details and impacted files
@@            Coverage Diff             @@
##             main        #115   +/-   ##
==========================================
  Coverage        ?   87.53096%           
==========================================
  Files           ?         467           
  Lines           ?       44005           
  Branches        ?           0           
==========================================
  Hits            ?       38518           
  Misses          ?        5487           
  Partials        ?           0           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

构建 Session / Memory 多后端回放一致性测试框架

1 participant